<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
	<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=iso-8859-1">
	<TITLE>SGE LAM Integration</TITLE>
	<META NAME="GENERATOR" CONTENT="StarOffice 6.0  (Solaris Sparc)">
	<META NAME="AUTHOR" CONTENT="Christopher.Duncan@Sun.COM">
	<META NAME="CREATED" CONTENT="20030729;9041800">
	<META NAME="CHANGED" CONTENT="20030729;9165000">
</HEAD>
<BODY LANG="en-US">
<H2>Sun ONE Grid Engine, Parallel Environment Integration for SGE
with LAM</H2>
<P ALIGN=CENTER>updated: <SDFIELD TYPE=DATETIME SDNUM="1023;1033;M/D/YY">7/29/03</SDFIELD></P>
<P>This is a test version of an SGE and LAM MPI intergration package.
Its only been minimally tested with MPI codes. Full LAM functionality
was not yet tested as of 7/22/03. This release is intended to get the
code into the hands of users for testing this initial release and
providing early feedback.</P>
<P> This code was tested against SGE 5.3p2 and LAM 6.5.9 on Solaris
8.<BR><BR>Download Here: <A HREF="sge-lam.tar">sge-lam.tar</A><BR><BR>For
updates and info regarding this code email:
<A HREF="mailto:christopher.duncan@sun.com?subject=SGE-LAM Integration">christopher.duncan@sun.com</A><BR><BR>This
code is provided AS-IS with no implied warranty or support.</P>
<H3><U>Tar Dist File Contents</U></H3>
<TABLE WIDTH=100% CELLPADDING=2 CELLSPACING=2>
	<TR VALIGN=TOP>
		<TD>
			<P><B>README.sge-lam</B></P>
		</TD>
		<TD>
			<P>Directions and info</P>
		</TD>
	</TR>
	<TR VALIGN=TOP>
		<TD>
			<P><B>qrsh-lam</B></P>
		</TD>
		<TD>
			<P>qrsh wrapper used for remote lamboot and for local lamd</P>
		</TD>
	</TR>
	<TR VALIGN=TOP>
		<TD>
			<P><B>sge-lam</B></P>
		</TD>
		<TD>
			<P>SGE compatible lamboot and lamhalt for use in start_proc_args
			and stop_proc_ags for and SGE PE</P>
		</TD>
	</TR>
</TABLE>
<HR>
<H3>Setup Instructions:</H3>
<OL>
	<LI><P>Install LAM MPI and SGE. This code was tested against SGE
	5.3p2 and LAM 6.5.9 and should work with later releases. It may work
	with earlier versions of SGE and LAM. <BR><B>NOTE</B>: make sure
	your shell startup env has both the LAM and SGE bin dirs in your
	path.</P>
	<LI><P>Install the 2 PERL executables: <CODE><B>qrsh-lam</B></CODE>,
	<B><CODE>sge-lam</CODE> </B>inside the LAM installation bin dir.
	Make sure they are executable.</P>
	<LI><P>Modify the variables: <CODE><B>LAMBINDIR</B></CODE> and
	<B><CODE>SGEBINDIR</CODE> </B>in <CODE><B>sge-lam</B></CODE> and<B>
	<CODE>qrsh-lam</CODE></B> to fit your site setup. The variables will
	depend on your installation of SGE and LAM.</P>
	<LI><P>Create an SGE PE that can be used to submit lam jobs. The
	following is an example assuming the scripts exist in
	<CODE>/usr/local/lam/bin</CODE>. You should replace the queue_list
	and slots with your site specific values.&nbsp;&nbsp;</P>
</OL>
<PRE>        % qconf -sp lammpi 
        pe_name lammpi
        queue_list hpc-v880.q polarbear.q
        slots 6
        user_lists NONE
        xuser_lists NONE
        start_proc_args /usr/local/lam/bin/sge-lam start
        stop_proc_args /usr/local/lam/bin/sge-lam stop
        allocation_rule $fill_up
        control_slaves TRUE
        job_is_first_task TRUE</PRE>
<OL START=5>
	<P><B>NOTE</B>: It is probably easiest to use the qmon GUI to create
	the PE.</P>
	<LI><P>Modify your LAM boot schema to use qrsh-lam. This is normally
	in the file <CODE>$LAMHOME/etc/lam-conf.lam</CODE>. You need to give
	a path to qrsh-lam and lamd for the boot schema. Normally this would
	be something like:<BR><BR>&nbsp;&nbsp; <CODE>lamd $inet_topo
	$debug</CODE><BR><BR>&nbsp;&nbsp; instead change this to (assuming
	your <CODE>LAMBINDIR</CODE> is
	<CODE>/usr/local/lam/bin</CODE>):<BR><BR><CODE>&nbsp;/usr/local/lam/bin/qrsh-lam
	local /usr/local/lam/bin/lamd $inet_topo $debug</CODE></P>
	<LI><P>With this PE setup users can submit jobs as normal and do not
	need to lamboot on their own. Users need only call mpirun for their
	MPI programs. Here is an example job:</P>
</OL>
<PRE>        % cat lamjob.csh
        #$ -cwd
        set path=(/usr/local/lam/bin $path)
        echo &quot;Starting my LAM MPI job&quot;
        mpirun C conn-60
        echo &quot;LAM MPI job done&quot;</PRE><P>
Using the C arg to mpirun is the easiest way to create a spanning MPI
job that uses all the allocated<BR>slots for MPI.</P>
<HR>
<H3>Current Issues:</H3>
<UL>
	<LI><P>A single user running multiple LAM jobs at once on the same
	nodes will have problems with accounting at minimum and may also
	improperly be halted when the first job exits. This limitation may
	be removed in future revs by using LAM_MPI_SOCKET_SUFFIX.</P>
	<LI><P>There have been some path issues with some shell setups and
	lamboot operating properly when qrsh;ing to remote nodes. This can
	normally be alleviated by ensuring your shell path includes the LAM
	bin dir.</P>
</UL>
<HR>
<H3>TODO:</H3>
<UL>
	<LI><P>Can we get rid of the <CODE>lam-conf.lam</CODE> changes by
	directly using hboot or something similar?</P>
	<LI><P>Modify <CODE>qrsh-lam</CODE> to be able to find its <CODE>qrsh</CODE>
	from <CODE>SGE_ROOT</CODE>. (for use in heterogenous clusters)</P>
	<LI><P>installer script to automate steps 1-6 after asking for
	<CODE>LAMHOME</CODE> and <CODE>SGE_ROOT</CODE>? also ask for a PE.</P>
</UL>
</BODY>
</HTML>